Return to Past and Current Projects
Decontaminating Human Judgments
For over half a century, psychologists have been struck by how poor people are at expressing their internal sensations, impressions, and evaluations via rating scales. When individuals make judgments, they are incapable of using an absolute rating scale, and instead rely on reference points from recent experience. This relativity of judgment limits the usefulness of responses provided by individuals to surveys, questionnaires, and evaluation forms. Fortunately, the cognitive processes that transform internal states to responses are not simply noisy, but rather are influenced by recent experience in a lawful manner.
We explore techniques to remove sequential dependencies, and thereby decontaminate a series of ratings to obtain more meaningful human judgments. In our formulation, decontamination is fundamentally a problem of inferring latent states (internal sensations) which, because of the relativity of judgment, have temporal dependencies. We propose a decontamination solution using a conditional random field with constraints motivated by psychological theories of relative judgment. Our exploration of decontamination models is supported by two experiments we conducted to obtain ground-truth rating data on a simple length estimation task. Our decontamination techniques yield an over 20% reduction in the error of human judgments.
Our current work involves expanding this approach from concrete perceptual judgments (e.g., line length) to more abstract judgments (e.g., affective judgment of images or art).
Students
Matt Wilder (Computer Science, University of Colorado)
Brittany Kos (Computer Science, University of Colorado)
Rob Lindsey (Computer Science, University of Colorado)
Ben Link (Computer Science, University of COlorado)
Collaborators
Matt Jones (Psychology, University of Colorado)
Michael Jones (Psychology, Indiana University)
Hal Pashler (Psychology, UCSD)